Interrater Agreement and Combining Ratings

Some behaviors such as smiles require human raters for their measurement. A model of the rating process is explored that assumes that the probability distribution of overt rating responses depends on which of several underlying or latent responses occurred. The ideal of theoretically identical raters is considered and also departures from such identity. Methods for parameter estimation and assessing goodness of fit of the model are presented. A test of the hypothesis of identical raters is provided. Simulated data are used to explore different measures of agreement, optimal numbers of raters, how the ratings from multiple raters should be used to arrive at a final score for subsequent analysis, and the consequences of departures from the basic assumptions of identical raters and constant underlying response probabilities. The results indicate that often using two or three raters to rate all of the data, assessing the quality of their ratings by assessing their pairwise correlations, and using their average rating rather than the ratings of a single primary rater will provide improved measurement of the underlying response. Simulations show that reduced rater sensitivity to response differences can be compensated for by using more raters. Methods are considered for using the model to select raters and to gauge which aspects of the rating process need additional training. Suggestions are offered for using the model in pilot rating of pilot data for deciding on the number of raters and the number of values on the rating scale.